2026-04-02 10:24:46,501 [ 261216 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:42, check_args_and_update_paths)
2026-04-02 10:24:46,501 [ 261216 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:86, check_args_and_update_paths)
2026-04-02 10:24:46,501 [ 261216 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:97, check_args_and_update_paths)
2026-04-02 10:24:46,502 [ 261216 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration  (runner:99, check_args_and_update_paths)
clickhouse_integration_tests_volume
Running pytest container as: 'docker run   --rm --name clickhouse_integration_tests_7xa0an --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=1e0b53d756cf -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e   -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e CLICKHOUSE_USE_OLD_ANALYZER=1 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="  -rfEps --run-id=2 --color=no --durations=0 test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors  -vvv" altinityinfra/integration-tests-runner:37a9815fd2fa '.
Start tests
============================= test session starts ==============================
platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /ClickHouse/tests/integration
configfile: pytest.ini
plugins: order-1.0.1, random-0.2, timeout-2.2.0, repeat-0.9.3, reportlog-0.4.0, xdist-3.5.0
timeout: 900.0s
timeout method: signal
timeout func_only: False
collecting ... collected 1 item

test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors FAILED [100%]

=================================== FAILURES ===================================
_________________ test_distributed_replica_max_ignored_errors __________________

    def test_distributed_replica_max_ignored_errors():
        settings = {
            "use_hedged_requests": 0,
            "load_balancing": "in_order",
            "prefer_localhost_replica": 0,
            "connect_timeout": 2,
            "receive_timeout": 2,
            "send_timeout": 2,
            "idle_connection_timeout": 2,
            "tcp_keep_alive_timeout": 2,
            "distributed_replica_max_ignored_errors": 0,
            "distributed_replica_error_half_life": 60,
        }
    
        # initiate connection (if started only this test)
        n2.query("SELECT * FROM dist", settings=settings)
        cluster.pause_container("n1")
    
        # n1 paused -- skipping, and increment error_count for n1
        # but the query succeeds, no need in query_and_get_error()
        n2.query("SELECT * FROM dist", settings=settings)
        # XXX: due to config reloading we need second time (sigh)
        n2.query("SELECT * FROM dist", settings=settings)
        # check error_count for n1
        assert (
            int(
                n2.query(
                    """
        SELECT errors_count FROM system.clusters
        WHERE cluster = 'replicas_cluster' AND host_name = 'n1'
        """,
                    settings=settings,
                )
            )
            == 1
        )
    
        cluster.unpause_container("n1")
        # still n2
>       assert get_node(n2, settings=settings) == "n2"
E       AssertionError: assert 'n3' == 'n2'
E         - n2
E         + n3

test_distributed_load_balancing/test.py:234: AssertionError
---------------------------- Captured stdout setup -----------------------------
Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml
Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml
Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml
------------------------------ Captured log setup ------------------------------
2026-04-02 10:24:52 [ 344 ] DEBUG : Command:['docker ps | wc -l'] (cluster.py:113, run_and_check)
2026-04-02 10:24:52 [ 344 ] DEBUG : Stdout:1 (cluster.py:121, run_and_check)
2026-04-02 10:24:52 [ 344 ] DEBUG : No running containers (conftest.py:95, cleanup_environment)
2026-04-02 10:24:52 [ 344 ] DEBUG : Pruning Docker networks (conftest.py:97, cleanup_environment)
2026-04-02 10:24:52 [ 344 ] DEBUG : Command:['docker network prune --force'] (cluster.py:113, run_and_check)
2026-04-02 10:24:52 [ 344 ] DEBUG : Command:["sysctl net.ipv4.ip_local_port_range='55000 65535'"] (cluster.py:113, run_and_check)
2026-04-02 10:24:52 [ 344 ] DEBUG : Stdout:net.ipv4.ip_local_port_range = 55000 65535 (cluster.py:121, run_and_check)
2026-04-02 10:24:52 [ 344 ] INFO : Running tests in /ClickHouse/tests/integration/test_distributed_load_balancing/test.py (cluster.py:2788, start)
2026-04-02 10:24:52 [ 344 ] DEBUG : Cluster start called. is_up=False (cluster.py:2795, start)
2026-04-02 10:24:53 [ 344 ] DEBUG : Docker networks for project roottestdistributedloadbalancing are NETWORK ID   NAME      DRIVER    SCOPE
 (cluster.py:855, print_all_docker_pieces)
2026-04-02 10:24:53 [ 344 ] DEBUG : Docker containers for project roottestdistributedloadbalancing are CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
 (cluster.py:863, print_all_docker_pieces)
2026-04-02 10:24:53 [ 344 ] DEBUG : Docker volumes for project roottestdistributedloadbalancing are DRIVER    VOLUME NAME
 (cluster.py:871, print_all_docker_pieces)
2026-04-02 10:24:53 [ 344 ] DEBUG : Cleanup called (cluster.py:876, cleanup)
2026-04-02 10:24:53 [ 344 ] DEBUG : Docker networks for project roottestdistributedloadbalancing are NETWORK ID   NAME      DRIVER    SCOPE
 (cluster.py:855, print_all_docker_pieces)
2026-04-02 10:24:53 [ 344 ] DEBUG : Docker containers for project roottestdistributedloadbalancing are CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
 (cluster.py:863, print_all_docker_pieces)
2026-04-02 10:24:53 [ 344 ] DEBUG : Docker volumes for project roottestdistributedloadbalancing are DRIVER    VOLUME NAME
 (cluster.py:871, print_all_docker_pieces)
2026-04-02 10:24:53 [ 344 ] DEBUG : Command:docker container list --all --filter name='^/roottestdistributedloadbalancing_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:113, run_and_check)
2026-04-02 10:24:53 [ 344 ] DEBUG : Unstopped containers: {} (cluster.py:890, cleanup)
2026-04-02 10:24:53 [ 344 ] DEBUG : No running containers for project: roottestdistributedloadbalancing (cluster.py:904, cleanup)
2026-04-02 10:24:53 [ 344 ] DEBUG : Trying to prune unused networks... (cluster.py:910, cleanup)
2026-04-02 10:24:53 [ 344 ] DEBUG : Trying to prune unused images... (cluster.py:926, cleanup)
2026-04-02 10:24:53 [ 344 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:113, run_and_check)
2026-04-02 10:24:53 [ 344 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:121, run_and_check)
2026-04-02 10:24:53 [ 344 ] DEBUG : Images pruned (cluster.py:929, cleanup)
2026-04-02 10:24:53 [ 344 ] DEBUG : Trying to prune unused volumes... (cluster.py:935, cleanup)
2026-04-02 10:24:53 [ 344 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:113, run_and_check)
2026-04-02 10:24:53 [ 344 ] DEBUG : Stdout:1 (cluster.py:121, run_and_check)
2026-04-02 10:24:53 [ 344 ] DEBUG : Setup directory for instance: n1 (cluster.py:2808, start)
2026-04-02 10:24:53 [ 344 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4534, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Create directory for common tests configuration (cluster.py:4539, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Copy common configuration from helpers (cluster.py:4559, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Generate and write macros file (cluster.py:4602, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_distributed_load_balancing/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n1/configs/config.d (cluster.py:4632, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n1/database (cluster.py:4649, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n1/logs (cluster.py:4660, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4746, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Setup directory for instance: n2 (cluster.py:2808, start)
2026-04-02 10:24:53 [ 344 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4534, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Create directory for common tests configuration (cluster.py:4539, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Copy common configuration from helpers (cluster.py:4559, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Generate and write macros file (cluster.py:4602, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_distributed_load_balancing/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n2/configs/config.d (cluster.py:4632, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n2/database (cluster.py:4649, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n2/logs (cluster.py:4660, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4746, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Setup directory for instance: n3 (cluster.py:2808, start)
2026-04-02 10:24:53 [ 344 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4534, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Create directory for common tests configuration (cluster.py:4539, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Copy common configuration from helpers (cluster.py:4559, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Generate and write macros file (cluster.py:4602, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_distributed_load_balancing/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n3/configs/config.d (cluster.py:4632, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n3/database (cluster.py:4649, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n3/logs (cluster.py:4660, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4746, create_dir)
2026-04-02 10:24:53 [ 344 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/.env (cluster.py:86, _create_env_file)
2026-04-02 10:24:53 [ 344 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file)
2026-04-02 10:24:53 [ 344 ] DEBUG : No config file found (config.py:28, find_config_file)
2026-04-02 10:24:53 [ 344 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file)
2026-04-02 10:24:53 [ 344 ] DEBUG : No config file found (config.py:28, find_config_file)
2026-04-02 10:24:53 [ 344 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 824 (connectionpool.py:547, _make_request)
2026-04-02 10:24:53 [ 344 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/.env', '--project-name', 'roottestdistributedloadbalancing', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n3/docker-compose.yml', 'pull'] (cluster.py:113, run_and_check)
2026-04-02 10:25:05 [ 344 ] DEBUG : Stderr:Pulling n1 ...  (cluster.py:123, run_and_check)
2026-04-02 10:25:05 [ 344 ] DEBUG : Stderr:Pulling n3 ...  (cluster.py:123, run_and_check)
2026-04-02 10:25:05 [ 344 ] DEBUG : Stderr:Pulling n2 ...  (cluster.py:123, run_and_check)
2026-04-02 10:25:05 [ 344 ] DEBUG : Stderr:Pulling n2 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check)
2026-04-02 10:25:05 [ 344 ] DEBUG : Stderr:Pulling n1 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check)
2026-04-02 10:25:05 [ 344 ] DEBUG : Stderr:Pulling n3 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check)
2026-04-02 10:25:05 [ 344 ] DEBUG : Stderr:Pulling n3 ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check)
2026-04-02 10:25:05 [ 344 ] DEBUG : Stderr:Pulling n3 ... status: image is up to date for a... (cluster.py:123, run_and_check)
2026-04-02 10:25:05 [ 344 ] DEBUG : Stderr:Pulling n1 ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check)
2026-04-02 10:25:05 [ 344 ] DEBUG : Stderr:Pulling n1 ... status: image is up to date for a... (cluster.py:123, run_and_check)
2026-04-02 10:25:05 [ 344 ] DEBUG : Stderr:Pulling n2 ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check)
2026-04-02 10:25:05 [ 344 ] DEBUG : Stderr:Pulling n2 ... status: image is up to date for a... (cluster.py:123, run_and_check)
2026-04-02 10:25:05 [ 344 ] DEBUG : Stderr:Pulling n3 ... done (cluster.py:123, run_and_check)
2026-04-02 10:25:05 [ 344 ] DEBUG : Stderr:Pulling n2 ... done (cluster.py:123, run_and_check)
2026-04-02 10:25:05 [ 344 ] DEBUG : Stderr:Pulling n1 ... done (cluster.py:123, run_and_check)
2026-04-02 10:25:05 [ 344 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/.env --project-name roottestdistributedloadbalancing --file /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n1/docker-compose.yml --file /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n2/docker-compose.yml --file /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n3/docker-compose.yml up -d --no-recreate') (cluster.py:3146, start)
2026-04-02 10:25:05 [ 344 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/.env', '--project-name', 'roottestdistributedloadbalancing', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n3/docker-compose.yml', 'up', '-d', '--no-recreate'] (cluster.py:113, run_and_check)
2026-04-02 10:25:09 [ 344 ] DEBUG : Stderr:Creating network "roottestdistributedloadbalancing_default" with the default driver (cluster.py:123, run_and_check)
2026-04-02 10:25:09 [ 344 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_n3_1 ...  (cluster.py:123, run_and_check)
2026-04-02 10:25:09 [ 344 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_n1_1 ...  (cluster.py:123, run_and_check)
2026-04-02 10:25:09 [ 344 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_n2_1 ...  (cluster.py:123, run_and_check)
2026-04-02 10:25:09 [ 344 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_n3_1 ... done (cluster.py:123, run_and_check)
2026-04-02 10:25:09 [ 344 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_n2_1 ... done (cluster.py:123, run_and_check)
2026-04-02 10:25:09 [ 344 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_n1_1 ... done (cluster.py:123, run_and_check)
2026-04-02 10:25:09 [ 344 ] DEBUG : ClickHouse instance created (cluster.py:3154, start)
2026-04-02 10:25:09 [ 344 ] DEBUG : get_instance_ip instance_name=n1 (cluster.py:2135, get_instance_ip)
2026-04-02 10:25:09 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_n1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:09 [ 344 ] DEBUG : Waiting for ClickHouse start in n1, ip: 172.16.1.4... (cluster.py:3161, start)
2026-04-02 10:25:09 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_n1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:09 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:09 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:09 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:09 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:09 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:09 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:09 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:10 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:10 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:10 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:10 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:10 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:10 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:10 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:10 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:10 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:11 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:11 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:11 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:11 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:11 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:11 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:11 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:11 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:12 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:12 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:12 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:12 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:12 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:12 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:12 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:12 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:13 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:13 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:13 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:13 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:13 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:13 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:13 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:13 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:14 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:14 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:14 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:14 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:14 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:14 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:14 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:14 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:14 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:15 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:15 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:15 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:15 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:15 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/09c717b7cc3f618f080af41678608b48d89caf348d4311709bc6319caa5a477f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:15 [ 344 ] DEBUG : ClickHouse n1 started (cluster.py:3165, start)
2026-04-02 10:25:15 [ 344 ] DEBUG : get_instance_ip instance_name=n2 (cluster.py:2135, get_instance_ip)
2026-04-02 10:25:15 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_n2_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:15 [ 344 ] DEBUG : Waiting for ClickHouse start in n2, ip: 172.16.1.3... (cluster.py:3161, start)
2026-04-02 10:25:15 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_n2_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:15 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/1a16a4b7cf39068ed0f013cd7a635ad636be8cb5f721cd4744a803a83e23dd8b/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:15 [ 344 ] DEBUG : ClickHouse n2 started (cluster.py:3165, start)
2026-04-02 10:25:15 [ 344 ] DEBUG : get_instance_ip instance_name=n3 (cluster.py:2135, get_instance_ip)
2026-04-02 10:25:15 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_n3_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:15 [ 344 ] DEBUG : Waiting for ClickHouse start in n3, ip: 172.16.1.2... (cluster.py:3161, start)
2026-04-02 10:25:15 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_n3_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:15 [ 344 ] DEBUG : http://localhost:None "GET /v1.42/containers/0c8903c1a0fa6468fd584615d0d874b0c7b34d71be0de50a89f2345d5b078105/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2026-04-02 10:25:15 [ 344 ] DEBUG : ClickHouse n3 started (cluster.py:3165, start)
2026-04-02 10:25:15 [ 344 ] DEBUG : Executing query DROP TABLE IF EXISTS data on n1 (cluster.py:3602, query)
2026-04-02 10:25:16 [ 344 ] DEBUG : Executing query DROP TABLE IF EXISTS dist on n1 (cluster.py:3602, query)
2026-04-02 10:25:16 [ 344 ] DEBUG : Executing query CREATE TABLE data (key Int) Engine=Memory() on n1 (cluster.py:3602, query)
2026-04-02 10:25:17 [ 344 ] DEBUG : Executing query 
        CREATE TABLE dist AS data
        Engine=Distributed(
            replicas_cluster,
            currentDatabase(),
            data)
         on n1 (cluster.py:3602, query)
2026-04-02 10:25:18 [ 344 ] DEBUG : Executing query 
        CREATE TABLE dist_priority AS data
        Engine=Distributed(
            replicas_priority_cluster,
            currentDatabase(),
            data)
         on n1 (cluster.py:3602, query)
2026-04-02 10:25:18 [ 344 ] DEBUG : Executing query 
        CREATE TABLE dist_priority_negative AS data
        Engine=Distributed(
            replicas_priority_negative_cluster,
            currentDatabase(),
            data)
         on n1 (cluster.py:3602, query)
2026-04-02 10:25:19 [ 344 ] DEBUG : Executing query DROP TABLE IF EXISTS data on n2 (cluster.py:3602, query)
2026-04-02 10:25:19 [ 344 ] DEBUG : Executing query DROP TABLE IF EXISTS dist on n2 (cluster.py:3602, query)
2026-04-02 10:25:20 [ 344 ] DEBUG : Executing query CREATE TABLE data (key Int) Engine=Memory() on n2 (cluster.py:3602, query)
2026-04-02 10:25:20 [ 344 ] DEBUG : Executing query 
        CREATE TABLE dist AS data
        Engine=Distributed(
            replicas_cluster,
            currentDatabase(),
            data)
         on n2 (cluster.py:3602, query)
2026-04-02 10:25:21 [ 344 ] DEBUG : Executing query 
        CREATE TABLE dist_priority AS data
        Engine=Distributed(
            replicas_priority_cluster,
            currentDatabase(),
            data)
         on n2 (cluster.py:3602, query)
2026-04-02 10:25:21 [ 344 ] DEBUG : Executing query 
        CREATE TABLE dist_priority_negative AS data
        Engine=Distributed(
            replicas_priority_negative_cluster,
            currentDatabase(),
            data)
         on n2 (cluster.py:3602, query)
2026-04-02 10:25:22 [ 344 ] DEBUG : Executing query DROP TABLE IF EXISTS data on n3 (cluster.py:3602, query)
2026-04-02 10:25:23 [ 344 ] DEBUG : Executing query DROP TABLE IF EXISTS dist on n3 (cluster.py:3602, query)
2026-04-02 10:25:24 [ 344 ] DEBUG : Executing query CREATE TABLE data (key Int) Engine=Memory() on n3 (cluster.py:3602, query)
2026-04-02 10:25:24 [ 344 ] DEBUG : Executing query 
        CREATE TABLE dist AS data
        Engine=Distributed(
            replicas_cluster,
            currentDatabase(),
            data)
         on n3 (cluster.py:3602, query)
2026-04-02 10:25:25 [ 344 ] DEBUG : Executing query 
        CREATE TABLE dist_priority AS data
        Engine=Distributed(
            replicas_priority_cluster,
            currentDatabase(),
            data)
         on n3 (cluster.py:3602, query)
2026-04-02 10:25:25 [ 344 ] DEBUG : Executing query 
        CREATE TABLE dist_priority_negative AS data
        Engine=Distributed(
            replicas_priority_negative_cluster,
            currentDatabase(),
            data)
         on n3 (cluster.py:3602, query)
2026-04-02 10:25:26 [ 344 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n1 (cluster.py:3602, query)
2026-04-02 10:25:27 [ 344 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n2 (cluster.py:3602, query)
2026-04-02 10:25:27 [ 344 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n3 (cluster.py:3602, query)
------------------------------ Captured log call -------------------------------
2026-04-02 10:25:28 [ 344 ] DEBUG : Executing query SELECT * FROM dist on n2 (cluster.py:3602, query)
2026-04-02 10:25:29 [ 344 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/.env', '--project-name', 'roottestdistributedloadbalancing', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n3/docker-compose.yml', 'pause', 'n1'] (cluster.py:113, run_and_check)
2026-04-02 10:25:30 [ 344 ] DEBUG : Stderr:Pausing roottestdistributedloadbalancing_n1_1 ...  (cluster.py:123, run_and_check)
2026-04-02 10:25:30 [ 344 ] DEBUG : Stderr:Pausing roottestdistributedloadbalancing_n1_1 ... done (cluster.py:123, run_and_check)
2026-04-02 10:25:30 [ 344 ] DEBUG : Executing query SELECT * FROM dist on n2 (cluster.py:3602, query)
2026-04-02 10:25:33 [ 344 ] DEBUG : Executing query SELECT * FROM dist on n2 (cluster.py:3602, query)
2026-04-02 10:25:33 [ 344 ] DEBUG : Executing query 
    SELECT errors_count FROM system.clusters
    WHERE cluster = 'replicas_cluster' AND host_name = 'n1'
     on n2 (cluster.py:3602, query)
2026-04-02 10:25:34 [ 344 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/.env', '--project-name', 'roottestdistributedloadbalancing', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n3/docker-compose.yml', 'unpause', 'n1'] (cluster.py:113, run_and_check)
2026-04-02 10:25:35 [ 344 ] DEBUG : Stderr:Unpausing roottestdistributedloadbalancing_n1_1 ...  (cluster.py:123, run_and_check)
2026-04-02 10:25:35 [ 344 ] DEBUG : Stderr:Unpausing roottestdistributedloadbalancing_n1_1 ... done (cluster.py:123, run_and_check)
2026-04-02 10:25:35 [ 344 ] DEBUG : Executing query SELECT * FROM dist on n2 (cluster.py:3602, query)
2026-04-02 10:25:36 [ 344 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3602, query)
2026-04-02 10:25:38 [ 344 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3602, query)
2026-04-02 10:25:40 [ 344 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3602, query)
2026-04-02 10:25:43 [ 344 ] DEBUG : Executing query 
    SELECT hostName()
    FROM cluster(shards_cluster, system.query_log)
    WHERE
        initial_query_id = '10a745a1e708475fa1f3fcf0ce9aaef9' AND
        is_initial_query = 0 AND
        type = 'QueryFinish'
    ORDER BY event_date DESC, event_time DESC
    LIMIT 1
     on n2 (cluster.py:3602, query)
---------------------------- Captured log teardown -----------------------------
2026-04-02 10:25:45 [ 344 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/.env', '--project-name', 'roottestdistributedloadbalancing', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n3/docker-compose.yml', 'stop', '--timeout', '20'] (cluster.py:113, run_and_check)
2026-04-02 10:25:55 [ 344 ] DEBUG : Stderr:Stopping roottestdistributedloadbalancing_n1_1 ...  (cluster.py:123, run_and_check)
2026-04-02 10:25:55 [ 344 ] DEBUG : Stderr:Stopping roottestdistributedloadbalancing_n2_1 ...  (cluster.py:123, run_and_check)
2026-04-02 10:25:55 [ 344 ] DEBUG : Stderr:Stopping roottestdistributedloadbalancing_n3_1 ...  (cluster.py:123, run_and_check)
2026-04-02 10:25:55 [ 344 ] DEBUG : Stderr:Stopping roottestdistributedloadbalancing_n2_1 ... done (cluster.py:123, run_and_check)
2026-04-02 10:25:55 [ 344 ] DEBUG : Stderr:Stopping roottestdistributedloadbalancing_n1_1 ... done (cluster.py:123, run_and_check)
2026-04-02 10:25:55 [ 344 ] DEBUG : Stderr:Stopping roottestdistributedloadbalancing_n3_1 ... done (cluster.py:123, run_and_check)
2026-04-02 10:25:55 [ 344 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:113, run_and_check)
2026-04-02 10:25:55 [ 344 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:113, run_and_check)
2026-04-02 10:25:55 [ 344 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:113, run_and_check)
2026-04-02 10:25:55 [ 344 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/.env', '--project-name', 'roottestdistributedloadbalancing', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_2/n3/docker-compose.yml', 'down', '--volumes'] (cluster.py:113, run_and_check)
2026-04-02 10:25:57 [ 344 ] DEBUG : Stderr:Removing roottestdistributedloadbalancing_n1_1 ...  (cluster.py:123, run_and_check)
2026-04-02 10:25:57 [ 344 ] DEBUG : Stderr:Removing roottestdistributedloadbalancing_n2_1 ...  (cluster.py:123, run_and_check)
2026-04-02 10:25:57 [ 344 ] DEBUG : Stderr:Removing roottestdistributedloadbalancing_n3_1 ...  (cluster.py:123, run_and_check)
2026-04-02 10:25:57 [ 344 ] DEBUG : Stderr:Removing roottestdistributedloadbalancing_n1_1 ... done (cluster.py:123, run_and_check)
2026-04-02 10:25:57 [ 344 ] DEBUG : Stderr:Removing roottestdistributedloadbalancing_n2_1 ... done (cluster.py:123, run_and_check)
2026-04-02 10:25:57 [ 344 ] DEBUG : Stderr:Removing roottestdistributedloadbalancing_n3_1 ... done (cluster.py:123, run_and_check)
2026-04-02 10:25:57 [ 344 ] DEBUG : Stderr:Removing network roottestdistributedloadbalancing_default (cluster.py:123, run_and_check)
2026-04-02 10:25:57 [ 344 ] DEBUG : Cleanup called (cluster.py:876, cleanup)
2026-04-02 10:25:57 [ 344 ] DEBUG : Docker networks for project roottestdistributedloadbalancing are NETWORK ID   NAME      DRIVER    SCOPE
 (cluster.py:855, print_all_docker_pieces)
2026-04-02 10:25:57 [ 344 ] DEBUG : Docker containers for project roottestdistributedloadbalancing are CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
 (cluster.py:863, print_all_docker_pieces)
2026-04-02 10:25:57 [ 344 ] DEBUG : Docker volumes for project roottestdistributedloadbalancing are DRIVER    VOLUME NAME
 (cluster.py:871, print_all_docker_pieces)
2026-04-02 10:25:57 [ 344 ] DEBUG : Command:docker container list --all --filter name='^/roottestdistributedloadbalancing_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:113, run_and_check)
2026-04-02 10:25:57 [ 344 ] DEBUG : Unstopped containers: {} (cluster.py:890, cleanup)
2026-04-02 10:25:57 [ 344 ] DEBUG : No running containers for project: roottestdistributedloadbalancing (cluster.py:904, cleanup)
2026-04-02 10:25:57 [ 344 ] DEBUG : Trying to prune unused networks... (cluster.py:910, cleanup)
2026-04-02 10:25:57 [ 344 ] DEBUG : Trying to prune unused images... (cluster.py:926, cleanup)
2026-04-02 10:25:57 [ 344 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:113, run_and_check)
2026-04-02 10:25:58 [ 344 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:121, run_and_check)
2026-04-02 10:25:58 [ 344 ] DEBUG : Images pruned (cluster.py:929, cleanup)
2026-04-02 10:25:58 [ 344 ] DEBUG : Trying to prune unused volumes... (cluster.py:935, cleanup)
2026-04-02 10:25:58 [ 344 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:113, run_and_check)
2026-04-02 10:25:58 [ 344 ] DEBUG : Stdout:1 (cluster.py:121, run_and_check)
============================== slowest durations ===============================
35.53s setup    test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors
15.77s call     test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors
14.00s teardown test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors
=========================== short test summary info ============================
FAILED test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors
========================= 1 failed in 65.48s (0:01:05) =========================
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 445, in <module>
    subprocess.check_call(cmd, shell=True)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'docker run   --rm --name clickhouse_integration_tests_7xa0an --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=1e0b53d756cf -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e   -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e CLICKHOUSE_USE_OLD_ANALYZER=1 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="  -rfEps --run-id=2 --color=no --durations=0 test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors  -vvv" altinityinfra/integration-tests-runner:37a9815fd2fa ' returned non-zero exit status 1.